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ABSTRACT 

Eight discriminators were identified and data vrere obtained from the 
records of 80 graduate students who attainea one o r four achievement lev- 
els at the conclusion of a beginning course in educational statistics. 
Although the internal discriminatory power of the set of eight measures 
was very high, estimates of the true power were discouragingly low. Two 

GRE measures were judged to be the best discriminators, but very poor when 

i 

considered alone or in combination* Prediction for the second achievement 
level appeared fairly strong, even for an external analysis. Linear as 
well as quadratic classification results are included. 

/ 
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Introduction 

The academic background of education and psychology graduate students 
enrolled in beginning statistical methods (or data analysis) courses is 
sometimes quite varied. In particular, their quantitative skills typically 
vary from those mastered in beginning high school mathematics to those mas- 
tered in the study of calculus. It might be desirable to restrict enroll- 
ment in statistical methods courses to those students who have attained a 
certain mastery level in mathematics. However f statistical methods courses 
are required of most doctoral students in education and psychology, regard- 
less of their mathematics mastery level. It might also be argued that mas- 
tery of mathematics beyond simple algebra is not requisite for the intended 
understanding to be gained in these courses. Mathematical maturity is but 
one student characteristic that may contribute to the variability of achieve- 
ment in graduate level statistical methods courses. Others might be age, 
past general academic achievement, past specific nonmathematical achieve- 
ment, and, possibly, personality characteristics. The purpose of this 
study was* to examine those characteristics of graduate students that poten- 
tially discriminate among groups of students in various levels of achieve- 
ment at the conclusion of an introductory course in educational statistical 
methods. 

M ethod 

Subjects 

The sample used in this study consisted of graduate students that had 
completed an introductory course in educational statistics at The Univer- 
sity of Georgia offered in the Department of Educational Psychology. Data 
were collected for classes of students who had enrolled in the course begin- 
ning with the Summer Quarter of 1970 and continuing through the Fall Quar- 
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ter of the 1974-75 academic year. Six classes, with mean size of 13.5 and 
range of 19-6, were taught by the same instructor (the first author). The 
content of the course remained fairly stable; approximately the first half 
was spent on the typical introductory descriptive methods, with the remain- 
ing time spent on simple correlation and regression. A total of 81 stu- 
dents was considered in this study. One student (non-degree) was excluded 
from the study because of incomplete records, reducing the total sample size 
to 80. As could be determined from the available records, a clear majority 
(64) of the students had undergraduate training for elementary and/or second- 
ary school teaching. The sample is characterized in more detail in Table 1. 

I 

i *" "■"""»"»••*•-•-" — — — — "« 
Insert Table 1 about here 

As is evident from examining Table 1, this fiourse^ the first in a three- 
Cc\yt6e eequence, appears to be primarily a service course for non-Education-, 
al Psychology graduate students— this also holds true for classes taught by 
other instructors. It migLt be mentioned that some students, particularly 
those in the fields of statistics and mathematics, start the sequence with " 
the second course. 
Variables 

Prior to data collection, potential discriminators of student achievement 
were specified. Files were then examined to determine the information avail- 
able for each student. Based on 87 cified and available information, thir- 
teen potential discriminators were selected: age of the student (AGE), 
scores on both the verbal (GREV) and quantitative portions (GREQ) of the 
Graduate Record Examination, scores on the common (NTEC) and the teaching 
area (NTET) portions of the National Teacher Examination, the number of hours 
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of undergraduate level counses in mathematics/statistics (UHMS) , the grade 
point average attained in those courses (UAMS) , the number of hours* of gra- 
duate level mathematics/statistics courses completed prior to the course 
in educational statistics (GHMS) , the grade point average achieved in those 
courses (GAMS) , the number of years' since the completion of the last mathe- 
matics/statistics course (YCMS) , the undergraduate grade point average 
(UGPA), the total number of graduate hours completed by the student prior 
to his taking the beginning statistics course (GHRS) , and graduate grade 
point average prior to the course (GGPA) * 

Since there were only a limited number of students for which four of the 
measures were available, these measures were excluded from subsequent analy- 
ses. The GUMS and GAMS measures were available for only niue of the 80 stu- 
dents; NTEC and NTET measures were available for only 39 and 35 students, res 
pectively. Thus, nine measures remained: AGE, GREV, GREQ, UHMS, UAMS, YCMS, 
UGPA, GHRS , and GGPA. 

One of four levels of end-of-course achievement was recorded for each 
student: A', B, C, or D. Achievement or grade levels fot the course were 
based on approximately eight quizzes, one test, and a' final examination; 
all three assessment methods were of the multiple-choice variety, and had 
very nearly the same number of items from class to class. Final course 
achievement levels were determined by a linear combination of z-s cores. 
Grade level distributions varied somewhat from class to class. For example, 
in one class approximately 58% was in the A-level and 17% in the C-level, 
while another class had only 6% in the A-level with 33% in the D-level* The 
numbers of students in the achievement levels were: A, 17; B, 33; C, 19; 
and D, 11. 

6 



Data Analyses 

Preliminary univariate analyses of variance were carried out to identify 
measures which did not show any premise of contributing (F<1.00) to multi- 
variate separation of the four end-of-course achievement level groups. All 
univariate F values for the nine remaining measures were greater than 1.95; 
hence all nine measures were retained for the final analyses. 

Data records for some students were not complete. Graduate Record Exami- 
nation scores were not available for 12 students and were estimated. Esti- 
mates for the incomplete data were based on the arithmetic mean on each GRE 
measure for all available scores across all four grade-ievels. For 13 stu- 
dents a YCMS measure could not be determined from the records since they 
had no undergraduate courses in mathematics or statistics. In these cases 
it was assumed that. they had such a course in their senior year of high 
school. Since these same 13 students had no undergraduate grade point aver- 
age in mathematics/statistics (UAMS) » an additional analysis was carried out 
using only the b7 students having the UAMS measure. 

In the analyses the condition of multivariate normality was assumed to 
be met; the condition of equality of the four population covariance ma- 
trices was assessed ".sing both a chi-square and an F statistic. When 
appropriate, separation among the four criterion populations in terms of 
mean vectors was assessed via Milks' lambda statistic. Values of a dis- 
tance measure between pairs of centroids were also obtained to verify the 
A, B t C, D "ordering" of the four grade levels, and to examine the centroid 
configuration. Such an ordering was used to detect "second-order" misclas- 
sificationa— -where a student was classified into a grade level nonadjacent 
to his actual level.: Also, an attempt was made to sort out the best and 



poorest discriminators, in terms of contribution to group separation* 

Classification procedures were used to assess the predictive accuracy 
of the total set and subsets, of discriminators. Both "internal" and "ex- 
ternal" classification results were considered. Results of an internal 
classification analysis are those obtained when measures for the students 
on whom the basic statistics (mean vectors and covariance matrices) were 
determined are resubstituted to obtain the values for the classification 
rules. In an external classification analysis statistics based on one set 
of students are used in classifying "new" students. The external classi- 
fi cat ion method used in this study is an extension of that proposed by 
Lachenbruch (1967). The procedure for the Lachenbruch method is as fol- 
lows; Compute the statistics for each of the possible total samples of 
size 79 obtained by omitting one student's vector of measures from the 
original total sample of 80, and record for each computation whether the 
omitted student is misclassif ied. 

/ 

The computer program used was one developed by the first author. This 
program yields linear and quadratic classification results — both internal 
and external analyses — as well as the usual values of means, covariance 
matrices, distances, test statistics, and indices for discrimination. 

Results 

The values of the statistics using p«8 and N*80 are reported in Table 
2. The F values are based on all 80 students, using estimated measures 

Insert Table 2 about here 

where necessary. 

Based on values of test statistics obtained, the condition of equality 
of the four population covariance matrices was judged untenable — the ob- 



8 



served value of a chi-square statistic (df - 108) was 151.40, p<.01; the 
value of an F statistic (df = 108, 5299) was 1.26, p<.05. Because of this 
conclusion, the appropriateness of the interpretation of Wilks 1 separation 
index (the value of which was A=0.297) may be questionable. Distances bet- 
ween pairs of groups based on a pooled covariance matrix verified the or- 
dering of the grade levels. The means for the four levels on the s'ingle 
significant linear discriminant function (LDF) were 9.07, 7.94, 7.55, and 
6.85, respectively. Distance-like measures ("likelihood distances 11 ) based 
on separate group covariance matrices also supported the ordering. The 
usual indices of relative predictor variable contribution — predictor-LDF 
correlations, or standardized LDF weights — must be interpreted with 
caution* In light of the difficulty of interpretation, all indicators — 
correlations, weights, univariate F-values — suggested that GREQ and GREV 
were the best predictors, and that GHRS and YCMS were the poorest. 

The unequal covariance structure suggested that a nonlinear classifica- 
tion rule be employed. Defining 

to be the square of the distance from the point in eight-space represent- 
ing student i (X^) to the point representing the means of the eight measures 
in group k (X^) , where is the sample (8x8) covariance matrix for group 
k, the following "quadratic" classification statistic was used: 

Pi 
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where p fc is the prior probability of membership in population k. This lat- 
ter expression represents the (posterior) probability of student i belong- 
ing to population k. A student is classified into that population from 
which the sample yields the largest value of P lk . The value of p fc used in 
this study is where N k is the size of the sample selected from popu- 

lation k, and N-EN, . 

k * 

The results of__the internal and external quadratic classification analy- 
ses are given in Table 3. Internal classification yielded a high proportion 
of overall correct classifications (0.838), whereas this proportion fell con- 
siderably with the external analysis (0.388). (The latter proportion is 
about what would be expected under! chance classification.) The only grade 
level for which predictive accuracy remained somew' ,A : respectable in the ex- 

a drop from 0,88 to 0*61. Since a linear 
rule — where the pooled sample covarl^ice matrix, S, replaces the matri- 
ces in the quadratic statistic, P lfc — is typically used in classification 
analyses, such results are also given. Linear classification (see Table 4) 
yielded poorer overall internal proportion of correct classifications (0.600), 

Insert Tables 3 and 4 about here 



ternal analysis was the B-level — 



but better overall external proportion (0.500). With the linear rule the 
smallest difference between internal and external results was for the A-level 
group, 0.76 to 0.71; th<p proportion for the B-level only dropped from 0.79 
to 0.67. Internal classification by the quadratic rule 4id not yield a sin- 
gle second-order misclassification; the linear rule yielded seven such mis- 
classifications. External classification by the quadratic and linear rules 
produced eight and nine second-order misclassifications, respectively. 
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Even though the GREQ and GREV measures appeared to be the best, internal 
quadratic classification yielded an overall proportion of only 0.450 for GREQ 
alone and 0.488 for^he two used in combination. External classifications 
using the two GKE measures alone yielded proportions about what would be ex- 
pected uy chance; when used in combination the proportion was slightly high- 
er than that expected by chance. When the UGPA measure was included with 
the two GRE measures, overall proportions were "6.612 and 0.500 for the in- 
ternal and external analyses, respectively. Again, relative respectability 
in terms of classification accuracy only held for the B-level students. 

An analysis involving the 6^ students for whom the grade point average at- 
tained in undergraduate level courses in mathematics/statistics (UAMS) was con- 
sidered did not yield drastically different results. The test statistics in- 
dicated unequal covariance structure (p<.01); the value of A was 0.444. Again, 

l 

GREQ and GREV appeared as the best\ discriminators, with GHjtS and AGE the poor- 
eat; the UAMS measure was near the middle of the nine measures in terms of 
relative importance. OveraU internal and external quadratic proportions of 
correct classifications were 0.925 and 0.433, respectively; the corresponding 
proportions obtained from the linear rule were .716 and .552. 
^ Discussion 
Perhaps the most striking finding was the drop in the proportion o cor- 
rect classifications from the internal analyses to the external analyses. 
That this was particularly true for the quadratic rule should not be too 
surprising, since with eight or nine predictor measures, the number of es- 
timated parameters is large relative to the sample sizes. The drop was not 
nearly as severe for the linear classification rule. Whereas the internal 
classification might be expected to overestimate the true proportion of cor- 
rect classifications, the external analysis yields an underestimation (Mich- 
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aelis, 1973). Even though the classification accuracy across all four grado 
levels is somewhat evasive — somewhere between 0,388 and 0*838 or between 
0.433 and 0.925 — the measures considered in, this stu^r might be expected 
to do fairly well for the higher grade levels. Further, an external analy- 



sis might be expected to yield better results if the number of predictor 
measures is reduced to include only the "better" ones, as was found in 

this study when three rather than all eight measures were used. This is 

/ 

presumably due to the fewer parameters that need be estimated — 24 with 

three predictors versus ]^ with eight predictors for a quadratic external analysis 



The results of this study might appear to support the v contention that 
GRE measures are good predictors of achievement in graduate school. How- 
ever, to make predictions on th£ basis of these measures, to the ex- 
elusion of others, may be quite hazardous. Predicted grade levels based 
on separate GRE measures tended to be lower for students in the higlp levels 
and higher for those in low levels. It ought to be mentioned that would 
the variability of the GRE measures be not as restricted as is typical for 
students already enrolled in graduate programs, the measures might appear 
as better predictors. 

The addition of undergraduate grade point average in mathematics/sta- 

\ \ 
tistics (UAMS) did not appreciably affect the predictive accuracy of the 

set of discriminators.* 9 "^ second-order misclassific^tion resulted for all 
four analyses — internal and external, and linear and quadratic — with 
the inclusion of UAMS; a student who was in the A-level was predicted to 
be in the D- level. The student^ A^Level performance was attributed to her 
tremendous effort; her UAMS measure was only 1.00. 

As mentioned previously, an internal analysis may. be expected to over- 
estimate the proportion of correct classifications; this is particularly 
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true for quadratic classification, as was found to be the case in this study, 
since covarianc^ matrices characterizing each sample are used* However, a 
linear rule in this study performed better in an external analysis* 

Lastly, it is of some interest to note the trends in the descriptive 
| data on the four groups of students (sec m ~ s ♦ For all measures save 
one, the trends were those that might b „ ,ected; grade level and .age, and 
' grade level and years since last mathematics/statistics course are inversely 
related, while grade level and GRE measures, grade level and grade point aver- 
ages, and grade level and number of undergraduate hours in undergraduate math- 
ematics/statistics courses are directly related* The one exception is the 
trend across the grade levels of the number of graduate hours completed 
prior to the statistics couise (GHRS); it appears that the B-level and par- 

ticularly the D-level students delay longer in taking the course. It turn- 

' i 

out that the GHRS measure contributed very little to the separation bet- 
ween the four groups. 
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Sex 

Male 48 
Female 32 

Degree Program 
Ed.D 28 
Ed.M. 27 
Ed.S. 12 
Ph.D. 11 
Non-Degree 2 



Table 1 
Sample Description 

Graduate Major 
^ Education 
Science 

Educational Psychology 
Reading 

Social Science 
Administration 
Special Education 
Mathematics 
Curriculum 
Vocational 
Other 
Non-Education 
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Table 3 
Frequencies and Proportions 
of Classifications 
(Quadratic Rule) 



Actual 
Grade-Level 



A 
B 
C 
D 



Internal 
Predicted Grade-Level 

A B C D Total 

15(.88) 2 0 0 17 

2 29G88) 2 0 33 

0 6 12(.63) 1 19 

0 0 0 11(1.0) 11 



Overall proportion of corract classifications ■ 0.838 



External 



Actual 
Grade-Level 



A 
B 
C 
D 



Predicted Grade-Level 

A BCD Total 

8(.47) 8 1 0 17 

4 20(.61) 9 0 33 

2 15 1(.05) 1 19 

0 5 4 2 (.18) 11 



Overall proportion of correct classifications • 0.388 

Note . Main diagonal entries indicate correct classifications; off-diag- 
onal entries indicate misclassifications . 



Table 4 
Frequencies and Proportions 
of Classifications 
(Linear Rule) 



Actual 



Grades-Level 



A 

B 
C 
D 



Internal 



Predicted Grade-Level 



A 

13(176) 

3 \ 

1 

0 
0 



B 



3 

26(.79) 
14 
5 



0 
3 

4(.21) 
1 



1 
1 
1 

5(.45) 



Overall proportion of correct classifications 



Total 

17 
33 
19 

11 » 
0.600 



Actual 
Grade- Level 



External . 
Predicted Grade-Level 
A B C 

A 12(.71) 4 =0 
B 5 22(.67) 4 

C 1 * 14 1(.05) 

DO 5 1 



D 

1 

2 
3 

5(.45) 



Overall proportion of correct classifications 



Total 
17 
33 
19 
11 
-0.500 



Note. Main diagonal entries indicate correct classifications; off-diagon- 
al entries indicate misclassifications. 
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